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Abstract 

The aim of the paper is to study the link between non additivity of some entropies and their 
boundedness. We propose an axiomatic construction of the entropy relying on the fact that entropy 
belongs to a group isomorphic to the usual additive group. This allows to show that the entropies 
that are additive with respect to the addition of the group for independent random variables 
are nonlinear transforms of the Renyi entropies, including the particular case of the Shannon 
entropy. As a particular example, we study as a group a bounded interval in which the addition 
is a generalization of the addition of velocities in special relativity. We show that Tsallis-Havrda- 
Charvat entropy is included in the family of entropies we define. Finally, a link is made between 
the approach developed in the paper and the theory of deformed logarithms. 
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I. MOTIVATIONS 



It is well known that the Shannon entropy of the couple of two independent random 



upl 



variables is the sum of their respective Shannon entropies [j| . This is also true for the Renyi 
entropy |8j. This fact comes from the well accepted axiom that the information given by 
two independent events should be the sum of their respective information. However, this 



result is not true for all the entropies, and hence for all the definitions of the in 



ormation 
with 



13 wit 



content of an event. For example, the Tsallis-Havrda-Charvat (THC) entropy H, 
parameter a behaves as 

H a (X, Y) = H a (X) + H a (Y) + (1 - a)H a (X)H a (Y) 

whenever X and Y are independent random variables. Therefore, the THC entropy is not 
additive. Furthermore, it appears that THC entropy is bounded, from below if a < 1 and 
from above if a > 1. 

Here, we try to generalize this observation by constructing entropies that are bounded 
by arbitrary bounds. An elegant way of doing this is to suppose that the entropy lives in 
a group (X, ©), X being for example a bounded subset of the real line, © being the law of 
composition. 

In the following, starting from the postulates that entropy should be ©-additive for 
independent variables and that it is a generalized mean of the information of individual 
events, we obtain the general form of the entropy over a group isomorphic to (R, +). We 
show as an example that the Tsallis-Havrda-Charvat entropy enters the framework presented 
here when the law © is a special case of the Lorentz law of velocities composition in special 
relativity. Furthermore, we establish the connection between the approach presented here 
and the approach developed by Kaniadakis and Naudts based on the notion of deformed 
logarithms. 

II. ENTROPY 

To obtain the general form of the entropy, we follow the lines of the derivation of the 
Renyi entropy in p. The basic ingredients to build Renyi entropy are the following: 

1. the information of a couple of independent individual events is the sum of their re- 
spective information, 



2. the information of a random variable is a mean information of the individual informa- 



3. the information is also additive for independent random variables 

If the mean used is the so-called Kolmogorov-Nagumo mean, it can be shown that the 
entropies satisfying the three axioms are the Renyi entropies (including of course Shannon 
entropy as a particular case) [6]. However, as mentionned in the introduction, some measures 
of information are not additive for a couple of independent random variables. We already 
gave the example of THC entropy. To generalize this idea, we are going to modify axiom 
1, and replace it by two others. The first one stipulates that the entropy as a physical 
quantity must belong to a group. Hence we assume that entropy lies in a set X which has 
a group structure when endowed with an addition that we write ©. The second axiom will 
be a modified version of axiom 1 in which the usual addition is replaced by the © addition. 
Hence the set of axioms we are going to work with is 

1. the information belongs to a group (X, ©), 

2. the information of a couple of individual events is the ©-sum of their respective infor- 
mation, 

3. the information of a random variable is a mean information of the individual informa- 
tion, 

4. the information is also ©-additive for independent random variables 

We now derive the general form of an entropy which satisfies this set of axioms. 

Let X be a discrete random variable and {pk}k its probability law, pk being the probability 
of an event labelled k. Let Y be another random variable, independent from X, whose 
probability law is denoted as {qk}k- Let I(p) be the information conveyed by a random 
variable of probability law {pk}k, and let l£ be the information conveyed by an individual 
event labelled k. 

According to the third axiom, the information I(p) for a full law is a mean of the individual 
information, and therefore there is a well behaved one-to-one and onto function / : X — ► X 
such that 



tion, 




(1) 
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We now seek admissible functions / that ensure that the information is ©-additive for 
independent random variables. Taking the fourth axiom into account, or 

I{pq) = I(p)®I{q), (2) 

requires that / satisfies 



(3) 



To go further, we suppose now that we work on a group (X, ffi) isomorph to (R, +). Let 
M ffi : X — > R be the associated morphism. We then have a © b = M ( ^ 1 (M e (a) + M®(&)) 
for any a and b in X. Using this in equation Q, and introducing g = M e o we get 

\ hi / \ k / \ I j 

This expression is true whatever p and q, and thus true for a uniform law q, that is If = I, V7. 
If we set J = M©(7) and J£ = M e (i£) , we get 



* (X>^ (J£ + J)) =9 (X>^ (4 



+ J, 



an equation that has been solved already for g (see for example |6( and references therein), 
the only solutions of which can be written as 

g-\z + J) = a(J)g-\z)+g-\j). (4) 

Since g~ 1 (z + J) = g~ l (J + z), we can write a(J)g~ 1 (z) + g~ x {J) = a(z)g' 1 (J) + g~ 1 {z) 
or (a(J) — l)/<7 -1 (J) = (a(z) — l)/g~ l {z) = 7. Inserting this into equation (J3J) leads to the 
following functional equations 

( ? - 1 (J + ^) = g-\j)+g-\z) for 7 = 
a( J + z) = a(J)a(z) for 7 7^ 0. 

The first case leads to g~ x (x) = cx whereas the second one imposes to function a to be an 
exponential. Then, we can write in this case 5 ,_1 (a;) 

= (2( 1 -°) ;c -l)/7. Therefore, since g = 
foM^ 1 , for 7 = we get f(x) = cM e (x) and for 7 ^ we obtain f(x) = (2^ a ^ M ^-l)/-f. 
Using the second axiom, the information of individual events is ©-additive for independent 
events, or 

47 = 11 © n (5) 



for individual independent events of probability pk and q\. Hence, for individuals events, I 
is a morphism between (R + *, x) and (X, ©): it sends the usual multiplication onto the © 
addition. We write M x this morphism to obtain 



I(p) = M^ 1 f^p fe M ffi (M x (l/p fc ))j for 7 = 0, 

^p fc 2 (1 ~ a)Mffi(Mx(1/pfc)) 



Hp) = M-^l Y^iog, 



Now, we have M* 



[X, ©) and M ffi : (X, 



for 7 7^ 0. 



L,+). Therefore M e o M x : 



(R + , x) — > (R, +) is the morphism sending multiplication onto the usual addition. In other 
words, M e o M x (p) oc log(p). Therefore, in the last equations, we choose M e o M x (p) = 
log 2 (p) and we end up with 



J(p) = M" 1 ^-^p fc log 2 (p fc )j for 7 = 0, 



1 — a 



log 2 



E 



for 7 7^ 0. 



If we impose the usual additivity, that is to say M e oc Id, we recover the Shannon or Renyi 
entropies. Note that this result is quite poor in itself since the entropies that are ©-additive 
are nonlinear transformations of the usual Renyi or Shannon entropies. As a consequence, 
the maximizing probability laws are the same since the nonlinear transform is a morphism 
and as such is one-to-one and onto. 



III. LORENTZIAN ADDITION AND ITS APPLICATION FOR BOUNDED EN- 
TROPIES 

In order to get bounded entropies, we can work into a bounded subset of the real line. 
A possibility relies on a generalization of the velocity composition law in special relativity. 
Let us consider the Lorentz addition 



h®h = : 



This law, in its symmetrical form h- = —h+, was used by L. Nottale in his theory of scale 



relativity 



3, 



to define a generalized composition of log scales. The asymmetrical form was 



developed by Dubrule&Graner [9|, and used by Dubrulle in his theory of finite size scaling 
We further used it to define finite size scale invariant stochastic processes Q]. 
The group considered here is h + ], ©) where h_ < and h + > 0. We recognize the 
velocity composition law in special relativity when — h- = h + is taken to be the speed of 
light. Furthermore, a semi-infinite case is obtained if one of the two bounds is infinite, or, 
the law becomes 

h®h = h + h- if K — ► +°° 

h®h = h + h- -7— if h- — ► -00 

h+ 

We then recover the usual addition by letting the remaining finite bound to reach infinity. 
Furthermore, the identity element of the group is since / ©0 = 0©J = /. To get the 
associated morphism, we assume differentiability of M®, write M e (J© J) = M e (/) + M e (J) 
and differentiate this expression with respect to J and then set J = to obtain the following 
differential equation 

il^(O) 

® [x) ~ (i-x/h_)(i-x/h + y 

whose solution reads 

M e (x) = M'JO)- ^log' 



h + — h- \ 1 — x/h. 

If the two bounds go to infinity, we easily check that the morphism is proportional to the 
identity, hence recovering the usual addition. By properly setting the free parameter M^(0), 
the inverses can be written as follows 



M~ l (x) = h + (l-2- x/h +) if h. 



—00 



= h-(l- 2- x/h -) if h+ — ► +00 

(ht —h_)x 

1 _ 2 h - h + 

= h - h + else 

h- - h+2 h - h + 

Note that for the semi-infinite cases it is implicitly assumed that 



for the case 7t_ — > —00, h+ > so that X =] — 00, h A 



for the case h + — >■ +00, h- < so that X =}h_, +oo[ 
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It is clear that in the unbounded case h± — > ±00 we recover either the Shannon entropy 
or the Renyi entropy, since when the bounds go to infinity, the morphism converges to the 
identity. In the case of the semi-infinite addition however, we get 



I(p) =h±\l- 



(l-a)h± 



k 

where we recover as a special case the Tsallis-Havrda-Charvat entropy for h±(l — a) = —1, 
since then the entropy reads 

The case a < 1 corresponds to an entropy bounded from below by h_ = l/(at — 1) < 
whereas a > 1 leads to an entropy bounded above by h + — l/(cx — 1) > 0. 

This derivation of the THC entropy clearly shows the interpretation of parameter a as a 
bound on the THC entropy. But as seen also from the derivation, a is really an additional 
free parameter in the analysis. It is not a physical parameter. Rather, the physics enters 
the entropy via the bounds h±. a is thus a parameter used to perform the right analysis of 
the physical system, as it is used in the multifractal analysis. 

In the bounded case, the entropy reads 



1 _ fV* „?) (l-oc)h + h_ 



h--h+ (EkP a k) (1 - a)h+h - 
and the three parameters can be let as they are. However, it should be pointed out again 
that the bounds are physical quantities, part of a thermodynamics that could be derived 
from this entropy. But in the same way we adopted to recover THC entropy in the semi- 
infinite case, we can eliminate one parameter by relating the bounds to a. For example, in 
order to get a simpler expression of the entropy, we can choose 

h+ - h- =1 _ a 

We then have the freedom to keep one of the bounds as a free parameter. For example, if 
we keep h+ as a free parameter, we obtain for the entropy the following expression 

ty^ - h 1 ~J2kPk 

{P) ~ + i-(i + (i-*)/> + )£^ 

an entropy bounded below by h- = h + /(l + (1 — a)h + ), and above by h + . 



IV. THE LINK WITH THE FRAMEWORK OF DEFORMED LOGARITHMS 
AND EXPONENTIALS 



In a series of papers 
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■ Il5lll6|. Kaniadakis and co-workers, as well as Naudts |13Lll4|. 
have developed the framework of entropy based on deformed logarithms. The idea in this 
framework is to consider that the entropy is an average quantity, and thus should write 



where Ag(pk) is the information associated to the single event labelled k, of probability pk- 
Parameter 9 rules the behavior of A. Function Aq is called a ^-deformed logarithm if it 
satisfies certain properties jlflj . An interesting interpretation has been given by Kaniadakis 
in Q| in terms of group theory. His point of view is similar to the idea we present here, 
and is based on the diagram in figure (JTJ. This diagram explicits the equivalence between 
the three groups of interest, namely (R, +), (R*' + , x) and (X, ®). The functions appearing 
on the arrows represent the morphisms linking the groups. The notations logg and exp^ 
represent the ^-deformed logarithm and exponential, respectively. 



exp 




R*>+, x ) 



(x, e ) 



FIG. 1: Equivalent groups used and the morphisms associated to link them. 



The difference between the approach presented here and that of Kaniadakis and Naudts, 
despite the conceptual approach, relies especially in the fact that space X can be different 
from K, and that we use the more general Kolmogorov-Nagumo generalized mean to define 
the entropy. 



S 



V. DISCUSSION 



The result in itself 



I{p) 
Hp) 



Mr 



i 1 (~ Yl Pk lo ^(Pk) J for 7 = 



M" 1 



a 



log 2 



. k 



for 7 / 



is not very impressive. This is due to the fact that the group considered is isomorphic 
to (R, +). It implies also that the maximising probability laws of the bounded entropies 
presented here are the same as the maximizing probability laws of the initial entropy More 
interesting is the fact that THC entropy is a special case of entropies that are non additive 
(in the usual sense) but satisfy 

I(pq)=I(p)+I(q)-^-I(p)I(q) 

for independent variables. This observation could open new twists in non-extensive statisti- 
cal mechanics. However, the relevance of the framework presented in this note remains to be 
proved on practical examples. Let us finally mention that the work presented here includes 



the results of Dukkipati et. al. 
using generalized mean. 



17| concerning the non-generalizability of Tsallis entropy 
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